dyncfg: require a ParameterScope on every Config by antiguru · Pull Request #38180 · MaterializeInc/materialize

antiguru · 2026-08-12T17:01:16Z

Motivation

The scope specifier introduced by the scoped feature flags work
(doc/developer/design/20260609_scoped_feature_flags.md) was an optional
.scoped(..) builder step on Config. A new dyncfg therefore defaulted
silently to environment-wide, and a config that is actually realized per replica
was easy to leave unannotated. That failure mode is invisible: the LaunchDarkly
sync loop never evaluates a replica context for an unannotated config, so a rule
targeting a replica or a size family is a no-op with no error anywhere.

Only 11 dyncfgs carried a scope.

Description

Make scope the fourth argument of Config::new and drop Config::scoped, so
every declaration has to make the choice, then annotate all 282 dyncfgs:
61 replica-local, 221 environment-wide, none cluster-coherent (no dyncfg
feeds OptimizerFeatures, which is the only place a cluster-scoped value is
resolved today).

The rule

Environment is the safe declaration and the right one when in doubt. It
preserves the unscoped behavior of a single value everywhere. A finer scope
enables divergence, so declaring one asserts that divergence is both safe and
useful for that config. Getting that wrong introduces a way to break an
environment that did not previously exist, while erring toward Environment
only forgoes a capability.

So where a config is realized is a necessary condition for a finer scope, not
a sufficient one. A config is declared Replica only if it is realized per
replica and it tunes that replica process's own resource usage (memory, CPU,
I/O, concurrency, timing) and cannot change what a dataflow produces, what
reaches durable state, or anything externally visible.

The 61 that qualify are one class: lgalloc, the memory limiter, the spill and
pager knobs, timely zero-copy, logical backpressure, hydration concurrency,
metrics scrape intervals, source client timeouts and snapshot parallelism. Size
and shape tuning, which is the use case the design doc cites.

Everything else is Environment, including:

Flags selecting a different implementation whose output-equivalence is an
assumption rather than a guarantee (the join and render paths, temporal
bucketing, the correction buffer, the upsert state representation). Where the
assumption holds, Environment costs nothing. Where it does not, a per-replica
rollout turns one bug into query results that differ by which replica served
them.
Anything reaching durable state, including the materialized view sink
paths and the persist-backed peek stash.
Anything externally visible: Kafka production and topic configuration,
the S3 COPY TO layout knobs, and the SQL Server change-table cleanup, which
deletes rows from the upstream database.
The persist configs as a class (96 of them). The persist client runs on
clusterd, but the same client code also runs in environmentd against the
same shards, and no replica scope can reach that copy. A rollout targeting
replicas would leave a shard's other writer on the old value indefinitely, so
the per-replica capability is one that could never be exercised uniformly.
Recorded in persist-client/src/cfg.rs rather than on each declaration.
balancerd's configs. balancerd syncs LaunchDarkly itself against a
balancer context keyed by provider, region and build version, with no
environment, cluster or replica beneath it. Environment names the coarsest
targeting granularity, not the environmentd process.
Values that must agree across a cluster's replicas, even when read on
clusterd. Flags deciding whether a definite error is emitted are the
sharp case: the error lands in the collection, so replicas disagreeing write
different contents rather than merely doing different amounts of work.

The full rule is written up on ParameterScope in src/dyncfg/src/lib.rs.

environmentd read sites, where the overrides have to be constructed first

Seven configs are read in environmentd but resolved per replica. They now read
through that replica's overrides, which is what makes their Replica
declaration real rather than decorative:

Controller::provision_replica builds a replica's TimelyConfig from
arrangement_exert_proportionality, enable_timely_zero_copy,
enable_timely_zero_copy_lgalloc and timely_zero_copy_limit.
ComputeController::add_replica_to_instance freezes
compute_replica_expiration_offset and
enable_arrangement_dictionary_compression_alpha into ReplicaConfig.
The controller's per-replica hydration interceptor enforces
compute_hydration_concurrency, described below.

Supporting changes: Config::get_with_overrides layers a replica's
ConfigUpdates over a ConfigSet (a mistyped override is logged and ignored
rather than panicking a controller path); the outer Controller keeps the pushed
override map and fans it out to the compute and storage controllers, so the
coordinator has a single push entry point; and the coordinator installs the map
before creating replicas during bootstrap, not only after, since the
provisioning-time values are frozen by then.

The replica's effective configuration

ReplicaTask owns a replica_dyncfg alongside the controller's environment-wide
dyncfg. It is seeded from the environment-wide set and then kept current from
the CreateInstance snapshot and each UpdateConfiguration delta passing
through the task, which Instance::specialize_command_for_replica has already
specialized for this replica. It therefore holds exactly what the replica itself
reads.

It has to be a set of its own rather than a clone of the controller's: a cloned
ConfigSet shares its values, so applying one replica's overrides to a clone
would overwrite the environment-wide configuration for everyone.

SequentialHydration is handed that set per call and holds no configuration of
its own, so it cannot read the environment-wide value by accident. This is the
shape worth repeating. Rather than documenting which set to read, arrange for
the wrong one not to be in scope.

Per-replica override lifetime

The coordinator re-pushes the override map only when the scoped configuration
itself changes, so dropping a replica does not rebuild it. The four holders of
the map (Controller::drop_replica, ComputeController::drop_replica, compute
Instance::remove_replica, storage Instance::drop_replica) now prune the
dropped replica themselves. Memory retention only: replica IDs come from a
monotonic sequence and are not reused, so a stale entry could never be picked up
by a later replica.

Verification

cargo check -p mz-environmentd -p mz-clusterd -p mz-balancerd --all-targets
is clean across all three binaries, bin/fmt applied.
New mz-dyncfg unit test for get_with_overrides: absent override, present
override, and a type-mismatched override falling back to the set's value.
New mz-compute-client unit tests. replica_dyncfg_tracks_config_commands
covers the seed, the CreateInstance snapshot, the UpdateConfiguration
delta, and the environment-wide set staying untouched.
hydration_concurrency_follows_supplied_config covers the interceptor
behaviorally: at a concurrency of one the second Schedule is held back, and
raising the concurrency in the supplied set releases it.
A mechanical audit parsing each Config::new call's balanced argument list
confirms all 282 declarations carry a scope. Two further audits are clean: no
environment-scoped config is read from a clusterd-side crate except the
documented deliberate cases, and every replica-scoped config has a
clusterd-side read or is one of the seven resolved in environmentd above.

Note on risk: #38206 removed the enable_scoped_system_parameters gate, so
scoped evaluation is always on and these annotations take effect as soon as a
targeting rule exists. An earlier revision of this description claimed the
change was inert until that flag was enabled.

Not covered by an automated test: the controller wiring for the six
provisioning-time and replica-creation-time configs. Exercising it needs a
ComputeController / orchestrator harness that does not exist today, so it is
verified by construction and by the get_with_overrides test underneath it.

🤖 Generated with Claude Code

https://claude.ai/code/session_01RZo8dwEyXbXwUq6wb7BkeU

antiguru

We should ensure that the optimizer overrides are scoped to their cluster.

antiguru · 2026-08-13T07:08:36Z

+        let replica_dyncfg = mz_dyncfgs::all_dyncfgs();
+        ConfigUpdates::from(dyncfg).apply(&replica_dyncfg);


Why do we specialize here instead of within the replica task, or where it's created?

Agreed, moved into the replica task in b23e77a.

ReplicaTask now owns a replica_dyncfg next to the environment-wide dyncfg, seeded at task creation and kept current by apply_config_command in the message loop, right where specialize_command already runs. SequentialHydration takes it as an argument to absorb_command / observe_response and holds no configuration of its own, so it cannot read the environment-wide value by accident. It reads the replica's or nothing.

Not at creation time, though: the task would then need the overrides plumbed into ReplicaClient::spawn as a second channel for something the command stream already carries, and it would go stale on the next push. Tracking the commands keeps one source of truth, and it is what the replica itself sees.

The test moved with the logic. replica_dyncfg_tracks_config_commands covers the seed, the CreateInstance snapshot, the UpdateConfiguration delta and the environment-wide set staying untouched. The interceptor's test is now behavioral rather than a getter check: at concurrency 1 the second Schedule is held back, and raising the concurrency in the supplied set releases it.

Generated by Claude Code

antiguru · 2026-08-13T07:11:27Z

+        self.replica_dyncfg_overrides = overrides
+            .values()
+            .flat_map(|replicas| replicas.iter())
+            .map(|(replica_id, updates)| (*replica_id, updates.clone()))
+            .collect();


Could this leak updates for replicas that are dropped?

Yes. Fixed in b23e77a.

The assignment replaces the whole map, so it does not accumulate across pushes, and I had assumed that was enough. It is not: the coordinator only calls push_replica_dyncfg_overrides when replica_scoped_config_changed, so dropping a replica does not rebuild the map. A dropped replica's entry survives until the next scoped-config change, and an environment that churns replicas while the scoped config sits still accumulates them indefinitely.

Now pruned in the drop path instead of relying on the coordinator to re-push: Controller::drop_replica, ComputeController::drop_replica, compute Instance::remove_replica and storage Instance::drop_replica, which is all four places holding the map.

No correctness consequence that I can find, since replica IDs come from a monotonic sequence and are not reused, so a stale entry could never be picked up by a later replica. It was memory retention only.

Generated by Claude Code

antiguru · 2026-08-13T07:26:10Z

+/// Environment-scoped because it is read twice, once by the optimizer in
+/// `environmentd` and once at render time on the replica, and the two reads must
+/// agree. It also changes what a subscribe emits, which replicas of one cluster
+/// may not disagree about.


What happens if they don't agree?

Nothing bad, which means my justification was wrong. Rewritten in b23e77a.

The two reads are asymmetric. In environmentd the flag gates whether optimize_dataflow_snapshot runs, which may clear with_snapshot on an import. On the replica, render.rs:279 computes if import.with_snapshot || !subscribe_snapshot_optimization { Include } else { Exclude }, so the replica-side read can only ever put a snapshot back, never take one away.

So both directions are safe. Flag on in environmentd and off on the replica means the snapshot is fetched anyway, correct but unoptimized. Flag off in environmentd and on at the replica means with_snapshot is still true, so the snapshot is included. Two replicas of one cluster disagreeing costs work, not correctness, since elision only happens where the optimizer proved the snapshot is not needed for the sink's output.

The real reason it is environment-scoped is the plan-time read: OptimizerConfig::from(&SystemVars) has no replica in scope, and no cluster either. Cluster would be the honest scope, but this is a plain OptimizerConfig field rather than an OptimizerFeature, and OptimizerFeatureOverrides is the only thing plan sites resolve cluster overrides through. persist_fast_path_order on the line above is in exactly the same position. Happy to file a follow-up for extending cluster resolution to the non-feature fields of OptimizerConfig if you think it is worth it.

Generated by Claude Code

antiguru · 2026-08-13T07:48:48Z

On the review summary, about the optimizer overrides being scoped to their cluster.

I checked every plan site, and the cluster-scoped optimizer overrides are resolved everywhere a cluster is in scope. cluster_scoped_optimizer_overrides(cluster_id) is applied in peek.rs (both paths), create_index.rs (all three), create_materialized_view.rs (all three), subscribe.rs (both), introspection.rs, coord.rs:3555 and frontend_peek.rs (both). The sites that do not resolve them are the ones with no cluster to resolve against: create_view.rs optimizes a view, which is not installed on a cluster.

Two gaps worth naming, neither introduced by this PR:

CatalogState MV rehydration (src/adapter/src/catalog/state.rs:1414) builds its OptimizerConfig with get_cluster(mv.cluster_id).config.features() but not the scoped layer, unlike every sequencer path above. A cluster-scoped override that changes an MV's plan would then have the rehydrated plan disagree with the one the sequencer produced. I have not traced how far that plan travels, so I am flagging rather than claiming a bug. Happy to dig in if useful.

Dyncfgs cannot be cluster-scoped at all today. That is why this PR annotates 276 dyncfgs as 184 replica and 92 environment, with zero cluster. Cluster resolution at plan time runs entirely through OptimizerFeatureOverrides, which covers OptimizerFeatures and nothing else, and no dyncfg feeds OptimizerFeatures. Two dyncfgs do reach the optimizer, as plain OptimizerConfig fields next to features: persist_fast_path_order and compute_subscribe_snapshot_optimization (optimize.rs:306-307). Both are conceptually cluster-coherent and both are forced to Environment, because declaring Cluster would be a silent no-op, exactly the failure mode this PR is about. Closing that gap means teaching the plan sites to resolve cluster overrides for the non-feature fields of OptimizerConfig too. Say the word and I will file it, or fold it into this PR if you would rather it not ship half-covered.

Generated by Claude Code

DAlperin

So, I've convinced myself the mechanism here is right. I wonder if we shouldn't get area owners to sanity check the scope for their flags though

martykulma

Thanks @antiguru ! A few of the source configurations should be environment - one of them is used by the dataflow and purification, and the others affect the emission of definite errors.

Other than that S&S lgtm!

martykulma · 2026-08-19T19:43:11Z

    Duration::from_secs(30),
    "Maximum amount of time we'll wait for SQL Server to report an LSN (in other words for \
    CDC to be fully enabled)",
+    ParameterScope::Replica,


This flag affects the behavior of both the source dataflow and purification, best make it Environment.

Changed to Environment in 925f650, and confirmed the purification read you're describing: mz_sql::pure reads it at pure.rs:1030, pure.rs:1614 and pure.rs:1964, alongside the dataflow read at sql_server/replication.rs:234. The purification read happens in environmentd with no replica in scope, so replica-local was wrong.

Added a note on the declaration recording that, so the environment scope does not later look like an oversight on a config that is also read on clusterd.

Generated by Claude Code

martykulma · 2026-08-19T20:31:02Z

    "pg_source_validate_timeline",
    true,
    "Whether to treat a timeline switch as a definite error",
+    ParameterScope::Replica,


this should be scoped to the environment.

Done in 925f650, along with sql_server_source_validate_restore_history below and kafka_low_watermark_check.

These three are the same rule and I had it written on ParameterScope without applying it here: a config whose value must agree across the replicas of a cluster stays environment-wide even when it is read on clusterd. Deciding whether a definite error is emitted is exactly that, since the error lands in the collection, so replicas disagreeing would write different contents rather than merely doing different amounts of work. Each declaration now says so.

One I left alone and would like your read on: pg_schema_validation_interval, immediately above these, is still Replica. It controls how often validation runs rather than whether the error is emitted, so both replicas still eventually emit it, but it does influence when a definite error is written. If that timing needs to be coherent too, it should move with the others.

Generated by Claude Code

martykulma · 2026-08-19T20:31:17Z

    "sql_server_source_validate_restore_history",
    true,
    "Whether to treat a restore history change as a definite error",
+    ParameterScope::Replica,


ditto for this one, should be environment scoped

martykulma · 2026-08-19T20:33:16Z

    true,
    "Whether to check the low watermark for Kafka sources and error if the start \
    offset/resume upper has been compacted away.",
+    ParameterScope::Replica,


this should be scoped to Environment

antiguru · 2026-08-20T09:02:17Z

Following up on @DAlperin's suggestion that area owners sanity-check their own flags, and @martykulma having done exactly that for Sources and Sinks.

The scope of a config follows from where its value is realized, so these annotations are only as good as my reading of each config's read sites. A wrong one is invisible: the sync loop evaluates a context that nothing consumes, so a targeting rule silently does nothing, with no error and no failing test. Worth a few minutes from each owner.

Where they live

Area	File	Total	Replica	Environment
Compute	`src/compute-types/src/dyncfgs.rs`	55	50	5
Sources and Sinks	`src/storage-types/src/dyncfgs.rs` (+ `sources/`)	47	37	10
Persist	`src/persist-client/src/cfg.rs` and friends, `src/txn-wal`	94	88	6
Adapter	`src/adapter-types/src/dyncfgs.rs`, `src/controller-types`	59	4	55
Shared	`src/dyncfg`, `src/metrics`, `src/dyncfg-file`	19	3	16
balancerd	`src/balancerd/src/dyncfgs.rs`	8	0	8

Sources and Sinks is done. The other five are unreviewed by their owners.

What to look for, in rough order of payoff

A Replica config whose value must agree across a cluster's replicas. This is the one with teeth. If replicas disagreeing would change what a collection contains, rather than only how much work it takes to produce it, it has to be Environment. Three of Marty's four corrections were this: flags deciding whether a definite error is emitted. Error semantics are durable, so divergence there is a correctness bug rather than wasted effort.
A Replica config that environmentd also reads. If environmentd resolves it for a specific replica, the read site has to resolve that replica's overrides, otherwise the declaration is a no-op. Seven such configs are listed in the PR description. If environmentd instead reads it during planning or purification, where no replica is in scope, it should be Environment. Marty's fourth correction was that case (sql_server_max_lsn_wait, read in mz_sql::pure).
An Environment config that could usefully be replica-local. Lowest stakes, since it costs a missed capability rather than a bug, but it is the case that motivated the PR: before it, 265 of these 282 were environment-wide by default rather than by decision.

The rule I applied is written up on ParameterScope in src/dyncfg/src/lib.rs if you want the reasoning rather than the summary.

One thing that changed under this PR

#38206 removed the enable_scoped_system_parameters gate, so scoped evaluation is always on. These annotations take effect as soon as a targeting rule exists for a parameter, rather than sitting dormant behind a flag. An earlier revision of the PR description claimed otherwise.

Nothing here blocks merge from my side. If an area turns out to want changes, they are one-line edits to the declaration.

Generated by Claude Code

aljoscha

Looks good from my side: one thought I had is whether we actually want to allow the persist configs to be per-replica. And many of the persist configs also affect the persist code running on envd, so does that work well together?

The scope specifier was an optional `.scoped(..)` builder step, so a new dyncfg silently defaulted to environment-wide and a config that is actually realized per replica was easy to leave unannotated. An unannotated replica-local config is a silent no-op for the scoped feature flags layer: LaunchDarkly never evaluates a replica context for it, so a rule targeting a replica or size family does nothing. Make `scope` the fourth argument of `Config::new` and drop `Config::scoped`, so every declaration has to make the choice, then annotate all 276 existing dyncfgs from their read sites: 183 replica-local, 93 environment-wide. The rule, written down on `ParameterScope`: a config is replica-local when its value is realized inside a `clusterd` process. That covers the compute and storage worker config sets, the persist client config set, and `mz_metrics`, all four of which the per-replica dyncfg push reaches. `environmentd` reading such a config for its own process is fine, and sees the environment-wide value. A config realized in `environmentd` is environment-wide even when its effect concerns one replica, and a config whose value must agree across the replicas of a cluster stays environment-wide even when it is read on `clusterd`. Some configs are read in `environmentd` but shipped to a specific replica, which is the case that needs overrides constructed before the read. Those are now resolved against the replica's scoped overrides: * `Controller::provision_replica` builds a replica's `TimelyConfig` from `arrangement_exert_proportionality`, `enable_timely_zero_copy`, `enable_timely_zero_copy_lgalloc` and `timely_zero_copy_limit`. * `ComputeController::add_replica_to_instance` freezes `compute_replica_expiration_offset` and `enable_arrangement_dictionary_compression_alpha` into `ReplicaConfig`. To serve them, `Config::get_with_overrides` layers a replica's `ConfigUpdates` over a `ConfigSet`, the outer `Controller` keeps the pushed override map and fans it out to the compute and storage controllers, and the coordinator installs the map before creating replicas at bootstrap rather than only after. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RZo8dwEyXbXwUq6wb7BkeU

The hydration limit is enforced in environmentd, by the controller's per-replica interceptor withholding `Schedule` commands, so declaring it replica-local only means something if the interceptor resolves its replica's override. Give `SequentialHydration` a config set of its own and feed it the `CreateInstance` and `UpdateConfiguration` commands it already absorbs. Those have been specialized for the replica by `Instance::specialize_command_for_replica`, so the set holds exactly what the replica itself reads. It has to be a set of its own rather than the controller's: a cloned `ConfigSet` shares its values, so applying the overrides to a clone would overwrite the environment-wide configuration for everyone. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RZo8dwEyXbXwUq6wb7BkeU

Name the coarsest targeting granularity, not the process. balancerd's configs are Environment because balancerd resolves them against its own regional LaunchDarkly context, with nothing finer beneath it, not because balancerd is lumped in with environmentd. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RZo8dwEyXbXwUq6wb7BkeU

The replica task is where per-replica command specialization already lives, and its configuration is not the hydration interceptor's private business. Give `ReplicaTask` a `replica_dyncfg` holding what its replica reads, kept current from the configuration commands passing through, and hand it to `SequentialHydration` on each call. The interceptor now holds no configuration of its own, so it cannot read the environment-wide value by accident. It reads the replica's or nothing. Prune per-replica dyncfg overrides when a replica is dropped. The coordinator re-pushes the override map only when the scoped configuration itself changes, so a dropped replica's entry was otherwise retained until the next such change, and an environment churning replicas accumulated them. Correct the justification on `compute_subscribe_snapshot_optimization`. The two reads need not agree. The replica-side read only ever puts a snapshot back, so a disagreement costs work rather than correctness. It is environment-scoped because the plan-time read has no replica in scope. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RZo8dwEyXbXwUq6wb7BkeU

A config that decides whether a definite error is emitted has to be coherent across the replicas of a cluster: the error lands in the collection, so replicas disagreeing would write different contents. That covers kafka_low_watermark_check, pg_source_validate_timeline and sql_server_source_validate_restore_history. sql_server_max_lsn_wait is environment-wide for a different reason. Source purification reads it in environmentd as well as the source dataflow on the replica, and the purification read has no replica in scope. Each carries a note on why an environment-scoped config is read on clusterd, so it does not read as an oversight. Also annotate two configs main added since the signature changed: subscribe_max_buffered_bytes, read on the coordinator loop, and enable_compute_error_distinct, whose doc already recorded that replicas rendering under different values write the same thing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RZo8dwEyXbXwUq6wb7BkeU

Persist configs were replica-local because the persist client config set runs on clusterd and the per-replica dyncfg push reaches it. That misses that the same client code also runs in environmentd, against the same shards, and no replica scope can reach that copy. A rollout targeting replicas would leave a shard's other writer on the old value indefinitely, so the per-replica capability is one that could never be exercised uniformly. Persist config also decides what lands in shared durable state, such as part sizes, encoding and compaction inputs. Per-replica divergence there means one shard written under several configurations depending on which process wrote, which is hard to attribute when something goes wrong. Declare the class environment-wide, and record the rule on ParameterScope and in the persist cfg module rather than on each of the 89 declarations. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RZo8dwEyXbXwUq6wb7BkeU

…safe A finer scope enables divergence, so declaring one asserts that divergence is both safe and useful. Being wrong that way adds a way to break an environment that did not previously exist, while erring toward the environment scope only forgoes a capability. Make that asymmetry the governing rule on ParameterScope, with realization site a necessary rather than sufficient condition, and re-audit the annotations against it. Keep Replica for configs tuning a replica process's own resource usage that cannot change what a dataflow produces, what reaches durable state, or anything externally visible: lgalloc, the memory limiter, the spill and pager knobs, timely zero-copy, backpressure, hydration concurrency, metrics scrape intervals, source client timing and snapshot parallelism. Move the rest to Environment. Flags selecting a different implementation whose output-equivalence is an assumption rather than a guarantee, since a per-replica rollout would turn one bug into query results differing by which replica served them. Anything reaching durable state, including the MV sink paths, the upsert state representation and the persist-backed peek stash that environmentd also reads. Anything externally visible, including Kafka production and topic configuration, the S3 copy layout knobs and the SQL Server change-table cleanup that deletes from the upstream database. And ore_overflowing_behavior, which sets process-global overflow behavior and is read in environmentd too. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RZo8dwEyXbXwUq6wb7BkeU

antiguru · 2026-08-24T08:41:14Z

@aljoscha your persist question turned into a rethink of the whole PR, so this supersedes the by-area table I posted above. Replica-local went from 94 to 61, environment-wide from 188 to 221.

Persist

Both of your points land, and the second is the decisive one.

There is no scope meaning "every persist client in this environment". Replica structurally excludes the environmentd client, which persists the catalog, storage collections and txn-wal against the same shards. A persist rollout targeted per replica could never reach that writer, so it would sit on the old value indefinitely while the replicas moved. That is not a per-replica capability, it is a rollout that cannot complete. Your first point stands alone too: persist config decides what lands in shared durable state, so per-replica divergence means one shard written under several configurations depending on which process happened to write, which is miserable to attribute during an incident.

All 89 replica-local persist configs are now Environment, joining the 7 already there. Recorded as a class rule in persist-client/src/cfg.rs and on ParameterScope rather than on 89 declarations.

The principle behind it, applied to everything else

Generalizing your point: Environment is the safe declaration. It preserves the unscoped behavior of a single value everywhere. A finer scope enables divergence, so declaring one asserts divergence is both safe and useful. Being wrong that way adds a way to break an environment that did not previously exist; being wrong toward Environment only forgoes a capability.

So realization site is now a necessary condition for a finer scope, not a sufficient one, and re-auditing against that moved 33 more configs:

Implementation switches whose output-equivalence is an assumption rather than a guarantee: the join paths (enable_compute_half_join2, enable_mz_join_core), the render paths, temporal bucketing, the correction buffer, the upsert state representation. Where the assumption holds Environment costs nothing. Where it does not, a per-replica rollout turns one bug into query results differing by which replica served them.
Reaching durable state: the MV sink paths, and the peek stash, which is persist-backed and read by environmentd too, so it is the same argument as persist.
Externally visible: Kafka sink batching and client.id, topic configuration, the S3 COPY TO layout knobs, and sql_server_cdc_cleanup_change_table*, which deletes rows from the upstream database.
ore_overflowing_behavior: sets process-global panic-vs-wrap on arithmetic overflow, and is read at compute-client/src/controller.rs:634 in environmentd as well as on clusterd.

@martykulma this also answers the one I left open for you. pg_schema_validation_interval is now Environment: I could not convince myself the timing of a definite error is safe to diverge, and under this rule that uncertainty decides it.

What stays replica-local

61 configs, one coherent class: lgalloc, the memory limiter, the spill and pager knobs, timely zero-copy, logical backpressure, hydration concurrency, metrics scrape intervals, source client timeouts and snapshot parallelism. Resource and shape tuning that cannot change output, which is the size-family use case the design doc cites.

@DAlperin the per-area sanity check is easier now: Persist and balancerd are uniformly environment-wide, so only Compute (33) and Sources and Sinks (21) have replica-local configs left to check, plus 4 in controller-types and 3 in mz_metrics.

The rule is written up on ParameterScope in src/dyncfg/src/lib.rs, and the PR description is updated.

Generated by Claude Code

antiguru · 2026-08-24T08:46:11Z

Every scope that changes in this PR, diffed against main by config name.

52 configs change scope, and all 52 go Environment → Replica. Nothing goes the other way. The 9 configs already declared Replica on main keep that scope. No config in this PR ends up with less targeting granularity than it has today, so the change is purely additive and cannot regress an existing LaunchDarkly rule.

(The re-audit I described earlier moved configs from Replica back to Environment relative to earlier revisions of this branch, not relative to main. Against main those are not changes at all.)

Compute (24)

compute_correction_v2_chain_proportionality, compute_correction_v2_chunk_size, compute_dataflow_max_inflight_bytes, compute_dataflow_max_inflight_bytes_cc, compute_flat_map_fuel, compute_hydration_concurrency, compute_logical_backpressure_inflight_slack, compute_logical_backpressure_max_retained_capabilities, compute_prometheus_introspection_scrape_interval, compute_replica_expiration_offset, compute_server_maintenance_interval, consolidating_vec_growth_dampener, enable_arrangement_dictionary_compression_alpha, enable_columnation_lgalloc, enable_compute_logical_backpressure, enable_lgalloc_eager_reclamation, lgalloc_background_interval, lgalloc_file_growth_dampener, lgalloc_local_buffer_bytes, lgalloc_slow_clear_bytes, linear_join_yielding, memory_limiter_burst_factor, memory_limiter_interval, memory_limiter_usage_bias

Replica provisioning, `controller-types` (4)

arrangement_exert_proportionality, enable_timely_zero_copy, enable_timely_zero_copy_lgalloc, timely_zero_copy_limit

`mz_metrics` (3)

mz_metrics_lgalloc_map_refresh_interval, mz_metrics_lgalloc_refresh_interval, mz_metrics_rusage_refresh_interval

Sources and Sinks (21)

aws_prefetch_sts_connect_timeout, enable_upsert_paged_spill, kafka_buffered_event_resize_threshold_elements, kafka_poll_max_wait, kafka_reconnect_backoff, kafka_reconnect_backoff_max, kafka_retry_backoff, kafka_retry_backoff_max, mysql_replication_heartbeat_interval, mysql_source_snapshot_exact_count_max_rows, mysql_source_snapshot_parallelism, mysql_source_snapshot_partition_min_rows, mysql_source_snapshot_partition_probed_prefixes_per_billion_rows, postgres_fetch_slot_resume_lsn_interval, sql_server_snapshot_progress_report_interval, storage_cluster_shutdown_grace_period, storage_rocksdb_cleanup_tries, storage_server_maintenance_interval, storage_suspend_and_restart_delay, storage_upsert_max_snapshot_batch_buffering, storage_upsert_prevent_snapshot_buffering

Unchanged, already `Replica` on `main` (9)

enable_lgalloc, enable_column_paged_batcher, enable_column_paged_batcher_spill, and the six column_paged_batcher_* knobs.

The other 227

Declared Environment, which is the value they already resolve to today. The declaration is now explicit rather than defaulted, which is the point of the PR, but no behavior changes for any of them.

Why these 52 and not more

Six of them are the reason the environmentd-side resolution work in this PR exists: arrangement_exert_proportionality, enable_timely_zero_copy, enable_timely_zero_copy_lgalloc and timely_zero_copy_limit are frozen into TimelyConfig at provisioning, and compute_replica_expiration_offset and enable_arrangement_dictionary_compression_alpha into ReplicaConfig at replica creation. compute_hydration_concurrency is the seventh, resolved through the replica task's config set. Without those, get_with_overrides and ReplicaTask::replica_dyncfg would have no consumer.

The rest are resource and shape tuning that cannot change what a dataflow produces: allocator, memory limiter, spill, backpressure, maintenance intervals, source client timeouts, snapshot parallelism.

Follow-ups for the configs that were considered for Replica and left Environment are filed per area: CPU-227 for compute (render and algorithm selection, the peek stash family, COPY TO layout) and SS-443 for Sources and Sinks (Kafka sink batching, source dataflow structure, upsert state representation, pg_schema_validation_interval). Each group is sized to be its own small PR, and each records what argument would need to be made to promote it.

Generated by Claude Code

antiguru · 2026-08-24T09:14:04Z

Thanks for your reviews! I merged this in a more conservative form than what it was in between, so other than the code changes to add the explicit scope, not much should've shifted. I filed some Linear issues to capture potential follow-up work.

…#38448) #38180 made the scope a required argument of `Config::new`. It landed after #38077's last CI run and before its squash merge, so `main` currently fails to compile `mz-persist` (five E0061 errors in `hedge.rs`, [test build 132566](https://buildkite.com/materialize/test/builds/132566)). The hedge configs are `Environment`-scoped like every other persist config, since the same client code runs in `environmentd` and `clusterd`. No functional change. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

antiguru commented Aug 13, 2026

View reviewed changes

antiguru requested review from DAlperin and def- August 13, 2026 08:50

antiguru marked this pull request as ready for review August 13, 2026 08:50

antiguru requested review from a team and aljoscha as code owners August 13, 2026 08:50

antiguru force-pushed the claude/dyncfg-scope-specifiers-tuvlsj branch 2 times, most recently from d94c317 to 64f5011 Compare August 19, 2026 13:01

DAlperin reviewed Aug 19, 2026

View reviewed changes

antiguru requested a review from martykulma August 19, 2026 17:55

martykulma approved these changes Aug 19, 2026

View reviewed changes

antiguru force-pushed the claude/dyncfg-scope-specifiers-tuvlsj branch from 64f5011 to 925f650 Compare August 20, 2026 08:51

aljoscha approved these changes Aug 20, 2026

View reviewed changes

claude added 7 commits August 24, 2026 08:30

antiguru force-pushed the claude/dyncfg-scope-specifiers-tuvlsj branch from 925f650 to 0082a05 Compare August 24, 2026 08:40

antiguru merged commit 5a4a36c into main Aug 24, 2026
83 checks passed

antiguru deleted the claude/dyncfg-scope-specifiers-tuvlsj branch August 24, 2026 09:13

ggevay mentioned this pull request Aug 25, 2026

Fix merge skew on main: declare a ParameterScope on the hedge dyncfgs #38448

Merged

antiguru mentioned this pull request Aug 25, 2026

compute: select the arrange merge batcher by dyncfg #38454

Merged

ggevay mentioned this pull request Aug 25, 2026

adapter: delete the staged cluster reconfiguration and direct replica paths #38103

Merged

		let replica_dyncfg = mz_dyncfgs::all_dyncfgs();
		ConfigUpdates::from(dyncfg).apply(&replica_dyncfg);

Uh oh!

Conversation

antiguru commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Description

The rule

environmentd read sites, where the overrides have to be constructed first

The replica's effective configuration

Per-replica override lifetime

Verification

Uh oh!

antiguru left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

antiguru commented Aug 13, 2026

Uh oh!

DAlperin left a comment

Choose a reason for hiding this comment

Uh oh!

martykulma left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

antiguru commented Aug 20, 2026

Where they live

What to look for, in rough order of payoff

One thing that changed under this PR

Uh oh!

aljoscha left a comment

Choose a reason for hiding this comment

Uh oh!

antiguru commented Aug 24, 2026

Persist

The principle behind it, applied to everything else

What stays replica-local

Uh oh!

antiguru commented Aug 24, 2026

Compute (24)

Replica provisioning, controller-types (4)

mz_metrics (3)

Sources and Sinks (21)

Unchanged, already Replica on main (9)

The other 227

Why these 52 and not more

Uh oh!

Uh oh!

antiguru commented Aug 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

antiguru commented Aug 12, 2026 •

edited

Loading

Replica provisioning, `controller-types` (4)

`mz_metrics` (3)

Unchanged, already `Replica` on `main` (9)